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ABSTRACT 

A common problem in astrophysics is determining how bright a source could be and still not 
be detected in an observation. Despite the simplicity with which the problem can be stated, 
the solution involves complicated statistical issues that require careful analysis. In contrast to 
the more familiar confidence bound, this concept has never been formally analyzed, leading to 
a great variety of often ad hoc solutions. Here we formulate and describe the problem in a 
self-consistent manner. Detection significance is usually defined by the acceptable proportion 
of false positives (background fluctuations that are claimed as detections, or the Type I error), 
and we invoke the complementary concept of false negatives (real sources that go undetected, or 
the Type II error), based on the statistical power of a test, to compute an upper limit to the 
detectable source intensity. To determine the minimum intensity that a source must have for 
it to be detected, we first define a detection threshold, and then compute the probabilities of 
detecting sources of various intensities at the given threshold. The intensity that corresponds 
to the specified Type II error probability defines that minimum intensity, and is identified as 
the upper limit. Thus, an upper limit is a characteristic of the detection procedure rather than 
the strength of any particular source. It should not be confused with confidence intervals or 
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other estimates of a source intensity. This is particularly important given the large number of 
catalogs that are being generated from increasingly sensitive surveys. We discuss, with examples, 
the differences between these upper limits and confidence bounds. Both measures are useful 
quantities that should be reported in order to extract the most science from catalogs, though 
they answer different statistical questions: an upper bound describes an inference range on the 
source intensity, while an upper limit calibrates the detection process. We provide a recipe for 
computing upper limits that applies to all detection algorithms. 

Subject headings: methods: data analysis - methods: statistical 

1. Introduction 

When a known or suspected source remains undetected at a prescribed statistical significance during an 
observation, it is customary to report the upper limit on its intensity. This limit is usually taken to mean the 
largest intrinsic intensity that a nominal source can have with an appreciable chance of going undetected. 
Or equivalently, it is the smallest intrinsic intensity it could have before its detection probability falls below 
a certain threshold. We emphasize that the upper limit is not meant to be an estimate or even a bound 
on the intensity of the source, but rather it is a quantification of the power of the detection procedure to 
detect weak sources. Thus, it is a measure that characterizes the detection process. The endpoints of the 
confidence interval, by contrast, provide ranges of possible values for the source intensity rather than quantify 
the sensitivity of the procedure. While the concept of upper limits has generally been well understood by 
astronomers as a form of censoring (Isobe, Feigelson, & Ncilson 1986) with an intrinsic connection to the 
dctcctability of a source (Avni et al. 1980), there has not been a statistically meaningful description that 
encapsulates its reliance on detectability as well as statistical significance. 

Moreover, while the term "upper limit" has been traditionally used in this manner, it has been also 
been used in cases where a formal detection process is not applied (e.g., when a source is known to exist at 
a given location because of a detection made in some other wavelength). In such cases, the upper edge of 
the confidence interval is derived and noted as the upper limit, regardless of the detectability of that source. 
In order to prevent confusion, we shall henceforth refer to the upper edge of a confidence interval as the 
"upper bound." Despite the intrinsic differences, numerous studies have described the computation of the 
upper bound as a proxy for the upper limit in increasingly sophisticated ways. The parameter confidence 
interval is a statistically well understood and is in common use. Kraft et al. (1991) and Marshall (1992) 
for example applied Poisson Bayesian-likelihood analysis to X-ray counts data to determine the credible 
range, and thus set an upper bound, on the source intensity. Feldman & Cousins (1998) recognized that the 
classical confidence interval at a given significance is not unique, and devised a scheme to determine intervals 
by comparing the likelihoods of obtaining the observed number of counts with the maximum likelihood 
estimate of the intensity and a nominal intensity; this procedure produces unique intervals where the lower 
edge; overlaps with zero when there are very few counts, and the upper edge stands as a proxy for an upper 
limit. Variations in the background were incorporated via a sophisticated Bayesian analysis by Weisskopf et 
al. (2007). 

The similarity of nomenclature between upper limits and upper bounds has led to considerable confusion 
in the literature on the nature of upper limits, how to compute them, and what type of data to use to do 
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so. Many techniques have been used to determine upper limits. It is not feasible to list all of thesejj but 
for the sake of definiteness, we list a few methods culled from the literature: the techniques range from 
using the root-mean-square deviations in the background to set the upper limit (Gilman et al. 1986, Ayres 
1999, Perez-Torres, et al. 2009), adopting the source detection threshold as the upper limit (Damiani et al. 
1997, Erdeve et al. 2009, Rudnick et al. 2009), computing the flux required to change a fit statistic value 
by a significant amount (Loewenstein et al. 2009), computing the p- value for the significance of a putative 
detection in the presence of background (Pease et al. 2006, Carrera et al. 2008), and identifying the upper 
limit with the parameter confidence bound (Schlegel & Petre 1993, Rolke et al., 2005, Hughes et al. 2007). 
Here, we seek to clarify these historically oft-used terms in a statistically rigorous way. 

Our goal here is to illustrate the difference between upper limits and upper bounds, and to develop a 
self-consistent description for the former that can be used with all extant detection techniques. Bounds and 
Limits describe answers to different statistical questions, and usually both should be reported in detection 
problems. We seek to clarify their respective usage here. We set out the requisite definitions and statistical 
foundations in Sj2] In !j3l we discuss the critical role played by the detection threshold in the definition of 
an upper limit and compare upper limits with upper bounds of confidence intervals. In 521 and Sj3]we use a 
simple Poisson detection problem as a running example to illustrate our methods. In fjJJ we apply them to 
a signal-to-noise detection problem. Finally, we summarize in SJS] 

2. Statistical Background 

Here we begin by describing our notation, and then discuss the nuances of the familiar concepts of 
confidence intervals, hypothesis testing, and statistical power. A glossary of the notation used is given in 
Table [H 

2.1. Description of the Problem 

Our study is carried out in the context of background-contaminated detection of point sources in photon 
counting detectors, as in X-ray astronomical data. We set up the problem for the case of uncomplicated source 
detection (i.e., ignoring source confusion, intrinsic background variations, and instrumental effects such as 
vignetting, detector efficiency, PSF structure, bad pixels, etc). However, the methodology we develop is 
sufficiently general to apply in complex situations. 

There is an important, subtle, and often overlooked distinction between an upper limit and the upper 
bound of a confidence interval, and the primary goal of this article is to illuminate this difference. The 
confidence interval is the result of inference on the source intensity, while the upper limit is a measure of 
the power of the detection process. We can precisely state this difference in the context of an example. 
Suppose that we have a typical case of a source detection problem, where counts are collected in a region 
containing a putative or possible source and are compared with counts from a source-free region that defines 
the background. If the source counts exceed the threshold for detection, the source is considered detected. 



^ An ADS query on astronomy abstracts, within the past year (excluding arXiv), containing "upper hmit", yields roughly 
two papers per day (759). A quick survey shows this term used in several disparate ways: some are upper bounds of confidence 
regions, often convolved with physics information to get the upper bound of a confidence region on (say) mass; others are clearly 
the theoretical power of a suggested test; yet others use 'upper limits' from previous work to obtain, e.g., line slopes. 
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Table 1. Symbols and notation 



Symbol 


Description 


ns 


counts observed in source area 


riB 


counts observed in background area 


As 


source intensity 


Ab 


background intensity 


As 


range in background intensity Xb 


rs 


exposure time 


TB 


exposure time for the background 


r 


ratio of background to source area 


S 


statistic for hypothesis test 


S* 


detection threshold value of statistic S 


n*s 


detection threshold value of statistic ns 


U 


upper limit 


a 


the maximum probability of false detection 


P 


the probability of a detection 


/?min 


the minimum probability of detection of a source with \s =U 


Pr(.) 


probability of 


n ~ /(.) 


denoting that n is sampled from the distribution /(.) 


Poisson(A) 


Poisson distribution with intensity A 


N'ili, a) 


Gaussian (i.e., normal) distribution with mean /i and variance cr^ 
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This detection threshold is usually determined by limiting the probability of a false detection. If the threshold 
were lower, there would be more false detections. Given this setup, we might ask how bright must a source be 
in order to ensure detection. Although statistically there are no guarantees, the upper limit is the minimum 
brightness that ensures a certain probability of detection. Critically, this value can be computed before 
the source counts are observed. It is based on two probabilities, (1) the probability of a false detection 
which determines the detection threshold, and (2) the minimum probability that a bright source is detected. 
Although the upper limit is primarily of interest when the observed counts are less than the detection 
threshold, it does not depend on the observed counts. This is in sharp contrast to a confidence interval for 
the source intensity, that is typically of the form "source intensity estimate plus or minus an error bar, " where 
the estimate and the error bars depend directly on the observed source counts in the putative source region. 
Of course, the functional form of the confidence interval may be more complicated than in this example, 
especially in low count settings, but any reasonable confidence interval depends on the source counts, unlike 
upper limits. The fact that upper limits do not depend on the source counts while confidence intervals do 
should not be viewed as an advantage of one quantity or the other. Rather it reflects their differing goals. 
Upper limits quantify the power of the detection procedure and confidence intervals describe likely values of 
the source intensity. These distinctions are highlighted with illustrative examples in §3.1.11 

To formalize discussion, suppose that a known source has an intrinsic intensity in a given passband of 
As and that the background intensity under the source is As. Further suppose that the source is observed 
for a duration ts and that ns counts are collected, and similarly, a separate measurement of the background 
could be made over a duration tb and n b counts are collected. If the background counts are collected in an 
area r times the source area@ we can relate the observed counts to the expected intensities, 

nB\{\B,r,TB) ^ Poisson(rrBAB) 
«s|(As, AbjTs) Poisson(^rs(A5 + As)^ , (1) 

in the background and source regions respectively, where ub and ns are independent. For simplicity, we 
begin by assuming that As is known. 

2.2. Confidence and Credible Intervals 

Confidence intervals give a set of values for the source intensity that are consistent with the observed 
data. They are typically part of the inference problem for the source intensity. The basic strategy is 
to compute an interval of parameter values so that on repeated observations a certain proportion of the 
intervals contain the true value of the parameter. It is in this "repeated-observation" sense that a classical 
confidence interval has a given probability of containing the true parameter. Bayesian credible intervals 
have a more direct probabilistic interpretation. They are computed by deriving the posterior probability 
distribution of the source intensity parameter given the observed counts and finding an interval with nominal 
probability of containing the true rate (see e.g., Loredo 1992, van Dyk et al. 2001, Kashyap et al. 2008). 



^ For clarity, we assume that the expected intensities are in units of counts per unit time and that the source and background 
counts are collected over pre-specified areas in an image. However, our analysis is not restricted to this scenario. The nominal 
background-to-source area ratio r could include differences in exposure duration and instrument effective area. Furthermore, 
the nominal exposure duration could also incorporate effective area, e.g., to have units [photons count" ^ cm^ s], which implies 
that the expected intensity Ag will have units [photons s"'^ cm~^]. Regardless of the units of Ag and As, the likelihood is 
determined by the Poisson distribution on the expected and observed counts as in Equation [T] 
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In summary, confidence intervals are frequentist in nature meaning that they are interpreted in terms of 
repeated observations of the source. Credible intervals, on the other hand, are Bayesian in nature meaning 
that they represent an interval of a certain posterior (or other Bayesian) probability. 

2.2.1. Confidence Intervals 

From a frequentist point of view randomness stems only from data collection — it is the data, not the 
parameters that are random. Often we use a 95% interval, but intervals may be at any level and we more 
generally refer to an L% confidence interval. Thus, the proper interpretation of a given interval is 

L% of experiments (i.e., observations) with intervals computed in this way will result in intervals 
that contains the true value of the source intensity. 

In frequentist terms, this means that in any given experiment one cannot know whether the true source 
intensity is contained in the interval but if the experiment is repeated a large number of times, about L% 
of the resulting intervals will contain the true value. Strictly speaking, the more colloquial understanding 
that there is an L% chance that the "source intensity is contained in the reported confidence interval," is 
incorrect. 

Put another way, a confidence interval for the source intensity gives values of As that are plausible given 
the observed counts. Suppose that A_b = 3 and that for each value of Xs, we construct an interval I(As) of 
possible values of the source counts that has at least an L% chance: Pr(ns G I(A5)|As, Xb,ts) > L%. Once 
the source count is observed and assuming \b is known, a confidence interval can be constructed as the set 
of values of As for which the observed count is contained in I{\s), 

{\s:ns^I(\s)}- (2) 

In repeated observations, at least L% of intervals computed in this way cover the true value of As. 

The frequency coverage of confidence intervals is illustrated in Figure [TJ where the confidence intervals 
of Garwood (1936) for a Poisson mean are plotted as boxes of width equal to the interval for various cases 
of observed counts (see van Dyk's discussion in Mandelkern, 2002). For given values of As and As the 
probability of the possible values of the observed counts can be computed using Equation[T] for each of these 
possible values, there will be a different confidence interval. Thus, the confidence intervals themselves have 
their own probabilities, which are represented as the heights of the boxes in Figure [TJ Because these are 95% 
confidence intervals, the cumulative heights of the boxes that contain the true value of As in their horizontal 
range must be at least 0.95. It is common practice to only report confidence intervals for detected sources 
so that only intervals corresponding to ns above some threshold are reported. Unfortunately this upsets 
the probability that the interval contains As upon repeated observations. Standard confidence intervals are 
designed to contain the true value of the parameter (say) 95% of time, i.e., in 95% of data sets. If some of 
the confidence intervals are taken away (i.e., are not reported because, e.g., the counts are too small), there 
is no reason to expect that 95% of those remaining will contain the true value of the parameter. This is 
because instead of summing over all values of ns to get a probability that exceeds 95%, we are summing 
only over those values greater than the detection threshold. This results in a form of Eddington bias and is 
discussed in detail in H3.41 
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Fig. 1. — Confidence Intervals for As, Computed for Cases with Different Source and Background Intensities. The 
true value of As is shown as a vertical dashed line and noted in the legend along with the true value of As. We 
assume that As is known exactly and adopt a nominal exposure ts = 1 and background scaling r = 10. Each box 
corresponds to a different value of ns- The horizontal width of each box denotes the width of the 95% confidence 
interval, and the height denotes the probability of observing that many counts given As- Top row: As = 1 and for 
As = 1, 3, 5 for the left, middle, and right columns respectively. Middle row. As for the top row, for As = 3. Bottom 
row. As for the top row, for As = 5. The figure illustrates that if the models are correctly specfied, very short 
intervals should be rare. Bayesian credible intervals (not shown) look similar, at least in high count scenarios. 
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2.2.2. Credible Intervals 



In a Bayesian setting, probability is used to quantify uncertainty in knowledge and in this regard 
parameters are typically viewed as random quantities. This distinction leads to a more intuitive interpretation 
of the credible interval. A credible interval at the L% level, for example, is any interval that contains the 
true value of the parameter L% of the time according to its posterior distribution. (See Park et al. 2008 for 
discussion on interval selection.) Thus, from a Bayesian perspective, it is proper to say that there is an L% 
chance that the source intensity is contained in the reported credible interval. The corresponding credible 
intervals look similar to the confidence intervals in Figure (TJ at least in high count scenariosjfl. 

So far we have considered a very simple problem with only one unknown parameter, Ag. The situation 
is more complicated if there are unknown nuisance parameters, such as As. In this case, frequency based 
intervals typically are constructed using asymptotic arguments and/or by conditioning on ancillary statistics 
that yield a conditional sampling distribution that does not depend on the nuisance parameter. Identifying 
ancillary statistics can be a subtle task and the resulting intervals may not be unique. Bayesian intervals, can 
be constructed using a simple and clear principle known as marginalization. For example, if As is unknown, 
the marginal posterior distribution of Xg is simply 



Credible intervals for A5 are computed just as before, but using the marginal posterior distribution. 



We emphasize that neither confidence nor credible intervals directly quantify the detection sensitivity 
of an experiment. To do this we consider the detection problem in detail, which from a statistical point of 
view is a test of the hypothesis that there is no source emission in the given energy bancH, i.e., a test of 
\s = 0. Formally, we test the null hypothesis that As = against the alternative hypothesis that As > 0. 
The test is conducted using a test statistic that we denote S. An obvious choice for S is the counts in the 
source region, ns; larger values of ns are indicative of a detection of a source, since they become increasingly 
less likely to have been obtained as a random fluctuation from the background. Other choices for S are 
the signal-to-noise ratio (as in the case of sliding-cell local-detect algorithms; see Q or the value of the 
correlation of a counts image with a basis function (as in the case of wavelet-based algorithms) or a suitably 
calibrated likelihood-ratio test statistic (as in the case of 7-ray detectors like Fermi; see Mattox et al. 1996). 



^In small count scenarios Bayesian credible intervals may not exhibit their nominal frequency coverage. They do however 
have the proper Bayesian posterior probability. 

*A popular frcquentist alternative to the marginal posterior distribution is the profile likelihood function (see Park et al. 
2008). Rather than averaging over nuisance parameters, the profile likelihood optimizes the likelihood over nuisance parameters 
for each value of the parameter of interest. 

^There is a close relationship between confidence intervals and hypothesis testing. If the interval includes zero, this indicates 
that there is a real possibility of no source emission above the background and if the source has not been otherwise detected 
there may be no source at all. Conversely, in Appendix |A] we discuss how a hypothesis test can be inverted to construct a 
confidence interval. 




(3) 



2.3. Hypothesis Testing and Power 
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The count in the source region is an example of a test statistic that is stochastically increasingj in As ■ For 
any fixed S* , Xb, ts, tb and r, the probability that the test statistic S is less than the threshold S* decreases 
as \s increases, i.e., Pr(iS < iS*|As, As, T5, r) decreases as As increases. We assume that S is stochastically 
increasing in As throughout 

Because larger values of S indicate a source, we need to determine how large S must be before we can 
declare a source detection. This is the done by limiting the probability of a false detection, also known as a 
Type I error. Thus, the detection threshold S* is the smallest value such that 

Pr(5 >5*|As = 0, As, rs, Ts, r) (4) 

where a is the maximum allowed probability of a false detectior[§ and we declare a detection if the observed 
value of S is strictly greater than S*: 

If S < S* we conclude there is insufRcient evidence to declare a source detection. If S > S* we 
conclude there is sufEcient evidence to declare a source detection. 



^ The term "stochastically increasing" means that there is a parameter (here As) that defines a distribution of observable 
values (here rig), and that all of the quantiles of 715 increase as Xg increases. There is no guarantee that at any single instance 
of observation, a higher Xg should lead to a higher ng. 

^ In principle, the test statistic is only required to have different distributions under the alternative and null hypotheses. 
For simplicity, we assume it tends to be larger under the alternative. 

* In the simple Poisson counts case, the probability of a Type I error given in Equation |4] is computed as, 

-TsXB(^rsXB)"'s e''-^B>^B(^rTBXB)"'B 



Pr(5 > 5*|As = 0,XB,Ts,TB,r) = 



1' ' 

Ig. 



where the summation is over the set of values of {n'g,n'g) such that S{n'g,n'g) > S* and we substitute A5 = into the mean 
ot ns given in Equation [T] Each term in the summation is a product of the likelihood of obtaining the specified counts in the 
absence of a source, given the background intensity and other observational parameters. In the simple case where S is the 
counts in the source region, ng, and Xg is known (i.e., ng is not measured), this reduces to 



Pr{S > S*\Xs = 0,Xb,ts) 



E ■ 



1 



s* 

E 



n's=0 S 

7(5* + l,rsAB) 
r(5* + 1) 



-^s^B (rsAB)"s 
r{n'g + 1) 

-^sAa (rsXB)"'s 
r(n' + 1) 



where 



7(5* + 1,tsXb) 



Jo 



is the incomplete gamma function (see Equations 8.350.1 and 8.352.1 of Gradshteyn & Ryzhik (1980)). In large count scenarios 
we may use continuous Gaussian distributions with their variances equal to their means in place of the discrete Poisson 
distributions in Equation [T] see Equation 1131 in In this case, we compute 



Pr(5>5*|As=0,As,rs 



,TB,r) = j 



exp 



(»'s-TS-^s)^ 



■rs>^B 



exp 



TTB^B 



^/2■KTsXl 



VSTrrrsAfl 



■ dn'gdn'g, 



where the integral is over the region of values of {n'g, n'g) such that S(n'g, n'g) > S* 
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Fig. 2. — a-Level Detection Threshold S* as a Function of the Background Intensity Xb, for the Given 
a Levels. Note that this is calculated assuming that the source intensity A5 = 0. The detection threshold 
increases with increasing Xb for a given a, and increases with decreasing a for a given Xb- 



We call S* the a-level detection threshold and sometimes write S*{a) to emphasize its dependance on a 
(see Figure [2]). Note that a is a bound on the probability of a Type I error; the actual probability of a 
Type I error is given by the probability on the left-hand side of Equation [H Due to the discrete nature of 
the Poisson distribution, the bound is generally not achieved and the actual probability of a Type-I error is 
less than a. 

Although its role in the definition of the detection threshold indicates that it is viewed as the more 
important concern, a false detection, also known as a "false positive," is not the only type of error. A false 
negative, or Type II error, occurs when a real source goes undetected (see Figure [3]). The probability of a 
false negative is quantified through the power of the test to detect a source as a function of its intensity. 



/3(As) =Pr(5>5*|As, As, Ts,TB, 



(5) 



Equation [5] gives the probability of a detection. For any Ag > 0, this is the power of the test or one 
minus the probability of a false negative]! For As = 0, Equation [5] gives the probability of a false detection 



® Here we use /3 to represent the power of the test, or one minus the probabihty of a Type II error. The statistical literature 
uses the notation /3 to denote either the Type II error (i.e., accepting the null hypothesis when it is false; e.g., Eadie et al. 
1971) or for the power of the test itself (as we have done here; e.g., Casella &; Berger 2002). As in the case of calculating a (see 
footnote [Sjl , we can calculate /9 as 



Pr(5 >S*\Xs,XB,rs,TB,r) = J2 



-r5(As+Aa)(^g(Ag+Afl))"S 



1' ' 



where again the summation is over the set of values of (n'g, n^) such that S(n'g, n'g) > S* 
is known, and ng is not measured, we find 



In the simple case where S = ng, 



Pv{S >S*\Xs,\B,rs) 



-rjs* + i,Ts{Xs + Xb)) 

r{ns + 1) 
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(cf. Equation |4]) and consequently, /3(0) < a. This reflects the trade-off in any detection algorithm: the 
compromise between minimizing the number of false detections against maximizing the number of true 
detections. That is, if the detection threshold is set low enough to detect weaker sources, the algorithm 
will also produce a larger number of false positives that are actually background fluctuations. Conversely, 
the more stringent the criterion for detection, the smaller the probability of detecting a real source (this is 
illustrated by the location of the threshold S* that defines both a and /3 in Figure [3]). Note that although 
our notation emphasizes the dependence of the power on As, it also depends on As, ts, tb, and r. 

The power calculation is shown for the simple Poisson case in Figure 21 where l3{Xs) is plotted for 
different instances of As and for different levels of the detection threshold S* . As expected, stronger sources 
are invariably detected. For a given source intensity, an increase in the background or a larger detection 
threshold (i.e., lower a) both cause the detection probability to decrease. In a typical observation, the 
background and the detection threshold are already known, and thus it is possible to state precisely the 
intensity As at which the source will be detected at a certain probability. We may set a certain minimum 
probability, /Smin of detecting a "bright" source by setting the exposure time long enough so that any source 
with intensity greater than a certain pre-specified cutoff has probability /^min or more of being detected. 
Conversely, we can determine how bright a source must be in order to have probability /^min or more of 
being detected with a given exposure time. This allows us to define an upper limit on the source intensity 
by setting a minimum probability of detecting the source. This latter calculation is the topic of § 13.11 and 
the basis of our definition of an upper limit. 

Power calculations are generally used to determine the minimum exposure time required to ensure a 
minimum probability of source detection (see Appendix |B]) . In ^we use them to construct upper limits. 

3. Upper Limits 

In this section, we develop a clear statistical definition of an upper limit that (i) is based on well- 
defined principles, (ii) depends only on the method of detection, (iii) does not depend on prior or outside 
knowledge about the source intensity, (iv) corresponds to precise probability statements, and (v) is internally 
self-consistent in that all values of the intensity below the upper limit are less likely to be detected at the 
specified Type-I error rate and values above are more likely to be detected. 

3.1. Definition 

In astronomy upper limits are inextricably bound to source detection: by an upper limit, an astronomer 
means 

The maximum intensity that a source can have without having at least a probabiUty of /^min of 
being detected under an a-level detection threshold. 



or conversely. 
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The smallest intensity that a source can have with at least a probability of /3niin of being detected 
under an a-level detection threshold. 

Unlike a confidence interval, the upper limit depends directly on the detection process and in particular on 
the maximum probability of a false detection and the minimum power of the test, that is on a and /Smin 
respectively. In this way, an upper limit incorporates both the probabilities of a Type I and a Type II error. 
Formally, we define the upper limit, U{a,j3^in) to be the smallest As such that 

Pr(5 > 5* (a) I As, As, rs,Ts,r) (6) 

Commonly used values for (3jnin throughout statistics are 0.8 and 0.9. If /3min ~ 1, W(a,/3min) represents the 
intensity of a source that is unlikely to go undetected, and we can conclude that an undetected source is 
unlikely to have intensity greater than U{a, /3min)- 

The simplest example occurs in the hypothetical situation when As is known to be zero and there is no 
background observation. In this case we set S — ns and note that Pr(ns > 0|As = 0,Xb = 0, ts) = so 
the detection threshold is zero counts and we declare a detection if there is even a single count. (Recall, we 
declare a detection only if S is strictly greater than S* .) The upper limit in this case is the smallest value of 
As with probability of detection greater than /Jmin- Figure [5] plots Pr(ns > 0|As, A^ — 0,ts) as a function 
of TsXs, thus giving W(a,/3inin) for any given rs and every value of /Smin- Notice the upper limit decreases 
in inverse proportion to rs. 

When As is greater than zero but well-determined and can be considered to be known, the detection 
threshold using iS = ns is given in Equation |4l With this threshold in hand we can determine the maximum 
intensity a source can have with significant probability of not producing a large enough fluctuation above 
the background for detection. This is the upper limit. 

In particular, W(a,/3rnin) is the largest value of As such that Pr(ns < 5*(a)|As, As, rs) > 1 — /?min- 
This is illustrated for three different values of /?niin (panels) and three different values of a (line types) in 
Figure m Notice that the upper limit increases as /3min increases and as a decreases. 



3.1.1. Illustrative Examples 

To illustrate the difference between confidence bounds and upper limits, in the context of a detection 
process, we consider two simple examples. The first is an extreme case where the background intensity is 
known to be identically zero, and even one count in the source region would be classified as a detection. In 
this case the upper limit is the smallest source intensity that can produce one count at a specified probability, 
e.g., a source with intensity of 5 generates one or more counts at a probability of w 99.7% (see ij3.ip . In 
contrast, if one count is seen in the source region, the upper bound of an equal-tail 99.7% interval on the 
source intensity is 8.9 (Gehrels 1986). Thus, while similar in magnitude, it can be seen that upper bounds 
and upper limits are different quantities, describing different concepts. 

Second, consider a more realistic case where the background is measured in a large region thought to be 
free of sources and scaled to the area covered by the source. Suppose that 800 counts are observed in an area 
400 times larger than the source area, and 3 counts are seen in the putative source region itself. The credible 
interval for the source intensity may be calculated at various significance levels ( ij2.21 see also van Dyk et al. 
2001), and for this case we find that the 68% credible interval with the lower bound at is [0, 2.1], and the 
99.7% interval is [0, 8.3]. But the question then arises as to whether the counts seen in the source region are 
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consistent with a fluctuation of the observed background or not. Since at a minimum 7 counts are needed for 
a detection at a probability of 0.997 (corresponding to a Gaussian-equivalent "Scr" detection), it is considered 
that the source is not detected. The question then becomes how bright the source would have to be in order 
to be detected with a certain probability. Since a source of intensity 5.7 would have a 50% probability of 
producing sufficient counts for a detection, this sets an upper limit U{a — 0.003, /3min — 0.5) ~ 5.7 counts 
on the undetected source's intensity (for a Type I error a = 0.003 and a Type II error (3 = 0.5; see ii3.1l) . 
Note that this limit is the same regardless of how many counts are actually seen within the source region, 
as expected from a quantity that calibrates the detection process. In contrast, the inference on the source 
intensity is always dependent on the number of observed source counts. 



3.2. Unknown Background Intensity 

So far our definition of an upper limit assumes that there are no unknown nuisance parameters, and in 
particular that As is known. Unfortunately, the probabilities in Equations 3] and [5] cannot be computed if 
As is unknown. In this section we describe several strategies that can be used in the more realistic situation 
when As is not known precisely. 

The most conservative procedure ensures that the detection probability of the upper limit is greater 
than /3min for any possible value of As- Generally speaking, the larger As is, the larger Xs must be in order to 
be detected with a given probability, and thus the larger the upper limit. Thus, a useful upper limit requires 
a finite range, A^, to be specified for A^. Given this range, a conservative upper limit can be defined as the 
smallest As that satisfies 

inf Pr(5 > 5* (a) I As, As, rs,Ts,r) > /3,nin. (7) 

(We use the term infimum (inf) rather than minimum to allow for the case when the minimum may be on 
the boundary of, but outside, the range of interest. It is the largest number that is smaller than all the 
numbers in the range. For instance, the minimum of the range {x > 0} is undefined, but the infimum is 0.) 
Unfortunately, unless the range of values is relatively precise, this upper limit will often be too large to 
be useful. 

In practice, there is better solution. The background count provides information on the likely values of 
Xb that should be used when computing the upper limit. In particular the distribution of As given hb can 
be computed using standard Bayesian procedure^ and used to evaluate the expected detection probability, 

/3(As) = Jpr{S>S*{a)\Xs,XB,Ts,TB,r)p{XB\nB,TB,r)dXB, (8) 

where S*{a) is the smallest value such that 

J Pr{S > S*{a)\Xs ^ 0,XB,Ts,TB,r)p{XB\nB,TB,r)dXB < a. (9) 

The upper limit is then computed as the smallest As that satisfies /3(As) > /3min- Unlike the upper limit 
described in ij3.11 these calculations require data, in particular ub- For this reason, we call the smallest As 

'^'^In the presence of a nuisance parameter, frequentist procedures are more involved and typically require conditioning on an an- 
cillary statistic, see, e.g., Appendix [C] In the Bayesian case, the posterior distribution, p(Afl|nfl, rg, r) oc p{As) p{nfl | As, rs , r), 
is the product of a prior distribution and the likelihood, normalized so that the posterior distribution integrates to 1. There 
are many choices of prior distributions available for As, ranging from uniform on As, to 7, to uniform in log(As) (see, e.g., van 
Dyk ct al. 2001). 
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that satisfies Equation [8] the background count conditional upper limit or bcc upper limit. An intermediate 
approach that is more practical than using Equation [7] but more conservative than using Equation [5] is to 
simply compute a high percentile of p(As|nB), perhaps its 95th percentile. The procedure for known Xb can 
then be used with this percentile treated as the known value of A^. This is a conservative strategy in that 
it assumes a nearly worst case scenario for the level of background contamination. 

As an illustration, suppose the uncertainty in given the observed background counts can be summa- 
rized in the posterior distribution plotted in the left panel of Figure [T] This is a gamma posterior distribution 
of the sort that typically arises when data are sampled from a Poisson distribution. Using this distribution, 
we can compute S* for any given value of a as the smallest value that satisfies Equation [9l the results are 
given for three values of a in the legend of Figure [7l We then use Equation [8] to compute f3{Xs) as plotted 
in the right panel of Figure [T] The upper limit can be computed for any /Smin using these curves just as in 
Figure El 

3.3. Confidence Intervals versus Upper Limits 

Although the form of a confidence interval makes it tempting to use its upper bound in place of an 
upper limit, this is misleading and blurs the distinction between the power of the detection procedure and 
the confidence with which the flux is measured. As an illustration we have computed the upper limit for 
each of the nine panels in Figure [T] (using the true values of Xb reported in each panel) . The results are 
plotted as solid vertical lines in Figure [H Notice that unlike the upper bound of the confidence interval the 
upper limit does not depend on ns and takes on values that only depend on Ab- Using the upper bound of 
the confidence interval sometimes overestimates and sometimes underestimates the upper limit. In all but 
one of the nine cases with the highest Xs/Xb (i-e.. As = 5, Xb = 1) the upper limit for A5 is larger than Xs- 
Of the nine cases, this is the one that is mostly likely to result is a source detection and is the only one with 
a probability of detection greater than /3min = 0.8. 

Alternatively, we can compute the value of /3min required for the upper bound of Garwood's confidence 
interval to be interpreted as an upper limit. Figure [9] does this for the three values of Xb used in the three 
columns of Figures [T] and [S) Consider how the upper bound of Garwood's confidence interval increases with 
ns in Figure [H Each of these upper bounds can be interpreted as an upper limit, but with an increasing 
minimum probability of a source detection, /3min- The three panels of Figure |8] plot how the required /Smin 
increases with ns for three values of As. Notice that a source with intensity equal to the upper bound of 
Garwood's confidence interval can have a detection probability as low as 20% or essentially as high as 100%. 
Thus, the upper bound does not calibrate the maximum intensity that a source can have with appreciable 
probability of going undetected in any meaningful way. 

3.4. Statistical Selection Bias 

As mentioned in §2.21 1 it is common practice to only report a confidence interval for detected sources. 
Selectively deciding when to report a confidence interval in this way can dramatically bias the coverage 
probability of the reported confidence interval. We call this bias a statistical selection bias. Note that 



A similar concern was raised by Feldman and Cousins (1998) who noticed that deciding between a one-sided and a two-sided 
confidence interval can bias the coverage probability of the resulting interval, if the decision is based on the observed data. 
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this is similar to the Eddington bias (Eddington 1913) that occurs when intensities are measured for sources 
close to the detection threshold. For sources whose intrinsic intensity is exactly equal to the detection 
threshold, the average of the intensities of the detections will be overestimated because downward statistical 
fluctuations result in non-detections and thus no intensity measurements. In extreme cases this selection 
bias can lead to a nominal 95% confidence interval having a true coverage rate of well below 25%, meaning 
that only a small percentage of intervals computed in this way actually contain Ag. As an illustration. 
Figure [To] plots the actual coverage of the nominal 95% intervals of Garwood (1936) for a Poisson mean 
when the confidence intervals are only reported if a source is detected with a — 0.05. These intervals are 
derived under the assumption that they will be reported regardless of the observed value of ng. Although 
alternative intervals could in principle be derived to have proper coverage when only reported for detected 
sources, judging from Figure [TU] such intervals would have to be wider than the intervals plotted in Figure [T] 
It is critical that if standard confidence intervals are reported they be reported regardless of the observed value 
of ns and regardless of whether a source is detectedrn 



3.5. The Detection Threshold as an Upper Limit 

As discussed in Section [l] the detection threshold is sometimes used as an upper limit. Under certain 
circumstances, this can be justified under our definition of an upper limit. Suppose that some invertible 
function f{S) can be used as an estimate of Xs and that for any As, the sampling distribution of f{S) is 
continuous with median equal to As. That is, 

Pr(/(5) > As|As,AB,Ts,TB,r) =0.5. (10) 

Because Equation [TUl holds for any A5, it holds for A5 = f{S*). That is, 

PT(^f{S) > f{S*)\Xs ^ f{S*),XB,Ts,TB,r) =0.5. (11) 

Inverting / and integrating over p{XB\nB, TB,r), we have 

J Pr(5>5*|As = /(5*),AB,Ts,rB,r)p(AB|nB,rB,r)dAs =0.5. (12) 

Comparing Equation [T^ with Equation [S] we see that f{S*) = Z-/(a, /3,nin = 0.5). Thus, if / is an identity 
function the detection threshold is an upper limit. Although the assumption that the sampling distribution 
of f{S) has median Xs for every As, is unrealistic in the Poisson case, it is quite reasonable with Gaussian 
statistics. Even if this assumption holds, f{S*) is a weak upper limit in that half the time a source with this 
intensity would go undetected and there is a significant chance that sources with intrinsic intensity larger 
than f{S*) would remain undetected. 

It should be emphasized that even when the detection threshold is used as an upper limit, it is not an 
"upper limit on the counts," but an upper limit on the intrinsic intensity of the source. The counts are an 
observed, not an unknown quantity. There is no need to compute upper bounds, error bars, or confidence 
intervals on known quantities. It is for the unknown source intensity that these measures of uncertainty are 
useful. 



Though this is usually not feasible when sources are detected via an automated detection algorithm such as celldetect 
or wavdetect. However, in many cases, source detectability is determined based on a pre-existing catalog, and in such cases, 
both Limits and Bounds should be reported in order to not introduce biases into later analyses. 
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3.6. Recipe 



Our analysis of upper limits and confidence interval assumes that the observables are photon counts 
that we model using the Poisson distribution. However, the machinery we have developed is applicable to 
any process that uses a significance-based detection threshold. Here, we briefly set out a general recipe to 
use in more complicated cases. For complex detection algorithms, some of the steps may require Monte 
Carlo methods. 

1. Define a probability model for the observable source and background dataset given the intrinsic source 
and background strengths. As and A_b, respectively. For the simple Poisson case, this is defined in 
Equation [TJ In many applications, these could be approximated using Gaussian distributions. It is 
typically required that a background data set be observed but in some cases Xb may be known a priori. 
The source could be a spectral line, or an extra model component in a spectrum, or possibly even more 
complex quantities that are not directly related to the intensity of a source. 

2. Define a test statistic S for measuring the strength of the possible source signal. In the simple Poisson 
case, we set S = ng. The "source" could be a spectral line or any extra model component in a 
spectrum, or a more complex quantity that is not related to the intensity of a source. 

3. Set the maximum probability of a false detection, a, and compute the corresponding a-level detection 
threshold, S* . Although S* depends on As, we can compute the expected S* by marginalizing over Xb 
using p{XB\nB) when As is not known exactly. Likewise, if As is defined as a function of several param- 
eters, the same marginalization procedure can be used to marginalize over any nuisance parameters. In 
this case, we typically marginalize over p(ri\nB) or perhaps p{ri\ns,nB), where rj is the set of nuisance 
parameters. In this regard, we are setting a to be a quantile of the posterior predictive distribution of 
S, under the constraint that As = 0, see Gelman et al. (1996) and Protassov et al. (2002). 

4. Compute the probability of detection, /3(As), for the adopted detection threshold S*. 

5. Define the minimum probability of detection at the upper limit, /3min- Traditionally /3niin = 0.5 has 
been used in conjunction with a ~ 0.003 in astronomical analysis (see i j3.5|) . 

6. Compute the smallest value of As such that /3(As) > /3min- This is the upper limit. 



We focus below on Signal-to-Noise (SNR) based detection at a single location as an example applica- 
tion. The SNR was the primary statistic used for detecting sources in high-energy astrophysics before the 
introduction of maximum-likelihood and wavelet-based methods. Typically, = 3 was used as the detection 
threshold, corresponding to a = 0.003 in the Gaussian regime. Here we apply the recipe in H'6.Q\ to derive 
an upper limit with SNR-based detection. Our methods can also be applied to more sophisticated detection 
algorithms such as sliding-cell detection methods such as celldetect (Harnden et al. 1984, Dobrzycki et al. 
2000, Calderwood et al. 2001), and wavelet-based detection methods such as pwdetect (Damiani et al. 1997), 
zhdetect (Vikhlinin et al. 1997), wavdetect (Freeman et al. 2002), etc. Implementation of our technique 
for these methods will vary in detail, and we leave these developments for future work. 

We begin with a Gaussian probability model for the source and background counts (Step 1 in §3.6|) 



4. Example: Signal to Noise Ratio 
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\{XsAb,ts) - A/" (^i = TsiXs + As), T - ^Ts{Xs + As)) , (13) 



where As and As are non- negative. We assume that the source is entirely contained within the source cell 
and that the PSF does not overlap the background cell. We can estimate Xb and As by setting ub and ns 
to their expectations (method of moments), as 

Xb = and As = . (14) 

rTB TS rTB 

The variance of As is 

M ^ , {Ts + rTB)XB .-.^s 

var(As) = h ^ (15) 

Ts ttstb 

which we can estimate by plugging in As and As as 

-^f^ . ns nB , . 

var(As) = — + (16) 

To use the SNR as a detection criterion, we define (Step 2 in §3.6p 

„ As rTBns - TsnB 



^far(As) V^^^B^s + r^ns 



(17) 



Step 3 in §3.61 says that the maximum probability of a false detection should be set and S* computed 
accordingly. Instead we adopt the standard detection threshold, S* = 3, used with the SNR and compute 
the corresponding a, 

"(As) = / p{n'g\XB,r,TB)p{n'g\Xs = 0,XB,Ts)dn'Bdn'g, (18) 
Jn 

where TZ is the region where S{n'g,n'g) > 5* = 3 (see footnote [8]) . For given values of As and r, the integral 
in Equation [18] can be easily evaluated via Monte Carlo. Alternatively, we can compute a by marginalizing 
over Xb if it is unknown, 

a{XB)p{XB\nB,r,TB)dXB, (19) 



where is the observed background count and a(As) is computed in Equation 1181 The probability of 
detection, /3(As) is computed (Step 4 in §3.6p by evaluating the same integral as in Equation [18] except that 
As is not set to zero. We can make the same substitution in Equation [19] if As is unknown. With /3min in 
hand (Step 5 in §3.6p . we can find the value of As such that /3(As) — /3min (Step 6 in H3.6^ . This is the upper 
limit. 

Figure [TT] illustrates the use of Equation [T8l to compure /3(As) for several values of As, with r ^ ts — 
tb = 1. The upper hmit is computed as the value of As such that (i{Xs) = /3min- The three panels of 
Figure [TT] report the resulting upper limits for /3niin = 0.5,0.9, and 0.95, respectively. 



5. Summary 

We have carefully considered the concept of upper limits in the context of undetected sources, and have 
developed a rigorous formalism to understand and express the concept. Despite its seeming simplicity, upper 
limits are not treated in a uniform fashion in astronomical literature, leading to considerable variations in 
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meaning and value. Wc formally define an upper limit to the source intensity as the maximum intensity it 
can have without exceeding a specified detection threshold at a given probability. This is defined by the 
statistical power of the detection algorithm. This is equivalent to defining it as the largest source intensity 
that remains undetected at the specified probability, and is defined by the probability of Type II error. Thus, 
if the detection probability is computed for a variety of source intensities, the upper limit is then identified 
by determining the intercept of the required probability with this curve. Thus, an upper limit is dependent 
only on the detection criterion, which is generally a function only of the background, and independent of the 
source counts. This is different from the upper bound (i.e, the upper edge of a confidence interval), which is 
obtained when the probability distribution of the source intensity is computed given that some counts are 
observed in the putative source region. We distinguish between the upper bound of the confidence interval 
and the upper limit of source detectability. Unlike a confidence interval (or Bayesian credible interval), an 
upper limit is a function of the detection procedure alone and does not necessarily depend on the observed 
source counts. 

The primary goals of this paper are to clearly define an upper limit, to sharpen the distinction between an 
upper limit and an upper bound, and to lay out a detailed procedure to compute the former for any detection 
process. In particular, we have shown how to compute upper limits for the simple Poisson case. We also 
provide a step-by-step procedure for deriving it when a simplified significance-based detection method is 
employed. To extract the most science from catalogs, we argue for using a consistent, statistically reasonable 
recipe of an upper limit being related to the statistical power of a test. In addition, we illustrate the peril of 
using an upper bound in place of an upper limit and of only reporting a frequentist confidence interval when 
a source is detected. Conversely, including confidence bounds, even for non-detections, is a way to avoid the 
Eddington bias and increase the scientific usefulness of large catalogs. 

We also describe a general recipe for calculating an upper limit for any well-defined detection algorithm. 
Briefiy, the detection threshold should be first defined based on an acceptable probability of a false detection 
(the a-level threshold), and an intensity that ensures that the source will be detected at a specifed probability 
(the /3-level detection probability) should be computed; this latter intensity is identified with the upper limit. 
We recommend that when upper limits are reported in the literature, both the corresponding a and j3 values 
should also be reported. 
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39073 (VLK, AS), NSF grants DMS 04-06085 and DMS 09-07522 (DvD). We acknowledge useful discussions 
with Rick Harnden, Frank Primini, Jeff Scargle, Tom Loredo, Tom Aldcroft, Paul Green, Jeremy Drake, and 
participants and organizers of the SAMSI/SaFeDe Program on Astrostatistcs. 

Appendix 

A. Constructing a Confidence Interval by Inverting a Hypothesis Test 

Here we discuss the relationship between confidence intervals and hypothesis tests and in particular how 
a hypothesis test can be used to construct a confidence interval. 

A confidence interval reports the set of values of the parameter that are consistent with the data. 
When this set includes Ag = it means that the data are consistent with no source and we expect the null 
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hypothesis not to be rejected and no source to be detected. There is a more formal relationship between 
confidence intervals and hypothesis testing and we can use a detection method to generate a confidence 
interval. Suppose that rather than testing the null hypothesis that Ag = 0, we are interested in testing the 
more general null hypothesis that A5 < A5, where Xg is any non- negative number. That is, we are interested 
in detecting only sources of at least a certain brightness. In this case, the detection threshold, S*{\*g), is 
defined as the smallest value such that 

Pr(5 >5*(AJ)|As = A|, As, rs, TB, r) < a. (Al) 

Given an observed value of S, we can construct the set of values Ag for which we cannot reject the null 
hypothesis that Ag < A^. This is a set of values of As that are consistent with the data and they form a 
100(1 — a)% confidence interval. This particular, confidence interval, however is of the form (a, +cxd): For 
any observed count there is a At, large enough so that we cannot reject the null hypothesis that A5 < A^. 
By reversing the null hypothesis to A5 > Ag we can obtain an interval of the form (0, a) and by setting up a 
two-sided test of the null hypothesis that A5 = A^ against the alternative hypothesis that A5 7^ Ag we can 
obtain an interval of the more common form (a, b). 

B. The Relationship between Upper Limits and the Power of the Test 

An upper limit turns around the usual use of the power of a test. Power is ordinarily used to determine 
the exposure time required to be sure that a source with intensity Asi„i„ or greater has at least probability 
/3min of being detected. That is, the smallest ts is found that satisfies Equation [6] for any As > Asmin and 
with Asinin fixed in advance. Thus, power is used to design an observation so that we have at least a certain 
probability of detecting a source of given brightness. With an upper limit on the other hand ts is fixed and 
Equation[n]is solved for A5. This is illustrated in Figure [T^ which plots ts versus As with fixed As, tb, r, a, 
and /3inin and shows what values of ts and As satisfy Equation[6]in the simple Poisson case. The shaded area 
above and to the right of the curve is where the detection probability exceeds /3min — 0.90. Thus the curves 
give the upper limit (on the horizontal scale) as a function of the exposure time. The upper limit generally 
decreases as the exposure time rs increases, but not monotonically. Due to the discrete nature of Poisson 
data, the threshold value S* changes in integer steps to allow for the inequality in Equation|3]to be satisfied. 
This behavior may be graphically illustrated by considering the sketch of the relevant quantities in Figure [3] 
As Ts increases, iS* increases in steps, causing the probability of false detection to abruptly fall and then 
smoothly increase to a. As the expected background in the source region increases, the upper curve shifts 
to the right, thereby increasing the shaded area that lies above the threshold value. However, when the area 
of the shaded region in the upper plot becomes larger than the tolerable probability of a Type I error, S* 
must be increased by one to reduce that probability. As rs increases, the lower curve remains stationary 
while S* is unchanging. At this stage, the upper limit, W(/3) is set as that value of As which ensures that 
the Type II error is /3 (see Equation [6]) , and thus slowly decreases as rs increases. When S* increases as a 
step function, the lower curve shifts to the right in order to maintain the same value of (3, and the upper 
limit abruptly increases. 

C. An Alternative Method for an Unknown Background Contamination Rate 

In the body of the article, we suggested conditioning on in order to effectively estimate As when 
it is unknown. A different strategy conditions instead on the total count ns + ns in order to remove Xb 
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from the model. This method is based on the simple probabilistic result that if X and Y are independent 
Poisson variables with means Xx and \y , respectively, then given X + Y , the variable X follows a binomial 
distribution. Applying this result to ng and ub with Poisson models given in Equatior[TJ we have 

ns I {ns + ns, As, As, r, rg, tb) - Binomial (ns + ub, — ts(As + Xb) \ ^ ^^^^ 

\ TsXs + (TS + rTB)\B J 

a binomial distribution with ns + jib independent counts each with probability ts{Xs + ^b)/ {ts^s + {ts + 
i'tb)Xb) of being a source count. Reparameterizing (A5, \b) via (,sXb = + A^, Equation IC II becomes 

ns I {ns + nB,Xs,XB,r,Ts,TB) ^ Binomial ( ns + nB, 7 — — — -, — ) , (C2) 

V is + rTB/TsJ 

which does not depend on the unknown background intensity. Here ^s — (As + Xb)/Xb which is equal to 
one if there is no source and grows larger for brighter sources. Because Equation IC2I does not depend on As 
it can be used for direct frequency based calculations even when As is unknown. In particular, a detection 
threshold can be computed based on a test of the null hypothesis that = 1, which is equivalent to As = 0. 
This is done using Equation 2] with S — nB and using distribution given in Equation IC2I with ^s = 1- Iii 
particular, we find the smallest S* such that Pr(ns > S*\ns + nB, S,s — Ij ts, tb) < a. With the detection 
threshold in hand, we can compute an upper limit for ^s using Equation [51 The upper limit is the smallest 
^s such that Pr(ns > S*\ns + nBT^SjfjTSiTB) > /3min- Unfortunately, this upper limit cannot be directly 
transformed into an upper limit for As without knowledge of As since As — Xb{^s ~ !)• 
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Fig. 3. — Illustration of Type I and Type II errors. A sketch of the probability distribution of test statistic 
S for specified values of the source and background intensities is shown for the simple case where S = ns 
and the background is known (see footnotes I8I9I) . The top panel depicts the probability Pr(iS = ngjAs = 
0, Xb = 2,ts — 1) and the bottom panel shows Pt{S — ns\\s — 5,Xb — 2, rg — 1). The vertical dashed line 
is a nominal detection threshold iS* that corresponds to a significance of a < 0.1, i.e., S* — 5. The Type I 
error, or the probability of a false positive, is shown by the shaded region to the right of the threshold. (The 
actual Type-I error for the adopted parameters is 0.05; values of S* less than 5 will cause the Type-I error 
to exceed the specified significance.) The Type II error is the probability of a false negative and is shown 
(for Xs — 5) by the shaded region to the left of the threshold in the lower panel. The detection probability 
of a source with intensity Xs — 5 is (3 = 0.7 for this choice of a, for the given background intensity, and for 
the exposure time. 
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Fig. 4. — Power of the Test, /3, to Detect a Source as a Function of Source Intensity, As, and Detection 
Threshold, S* . The curves are calculated for different values of the background intensity (the same values 
as in Figure [T|), As = 1 (left). As = 3 (middle), and A_b = 5 (right). The individual curves show l3{Xs) 
for different S* , each of which corresponds to a different bound on the probability of a Type I error, a, 
see Figure [2] The solid, dashed, and dash-dotted lines correspond to increasing detection thresholds, and 
decreasing values of a. As one would expect, /3 is higher for larger A5 and lower A^, i.e., if the source is 
stronger or the background is weaker, it is easier to detect. 




Fig. 5. — Upper Limit with No Background Contamination. The figure plots Pi{ns > 0|As,Ab = 0,rs) as 
a function of T5A5, thus giving W(a,/3min) for any given T5 and every value of /3min- For example, reading 
across the line plotted at (3 — 0.8, gives TsU{a = 0.05, /3,„in = 0.8) — 1.6, which can be solved for the upper 
limit for any value of Tg . Notice the upper limit decreases in inverse proportion to Tg . 
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Fig. 6. — Computing Upper Limits Based on the Probability of Detecting a Source. The figure iUustrates 
how upper hmits may be defined for different probabihties of source detection under a given detection 
threshold. The curves correspond to /3(As) for different values of S* and a: 5 and 0.1 (solid), 6 and 0.05 
(dashed), and 8 and 0.01 (dash-dotted) and were all computed with Xb = 3, as in the middle panel of 
Figure m Upper limits are computed by first adopting an acceptable probability for a source detection, and 
then computing the intercept on A5 of the P{Xs) curves. The panels show the value of the upper limits for 
the different values of S* for /3niin = 0.5 (top), /^min = 0.9 (middle), and /3min = 0.95 (bottom). 
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Fig. 7. — Upper Limit with Unknown Background Intensity. The left panel plots a Bayesian posterior 
distribution for As, p{\B\nB,TB,i') oc A^^e"^^/^-^, that is used in Equations [8] and |9] to compute (3, the 
expected detection probability, as a function of As with ts = tb = r = 1. The right panel plots l3{Xs) for 
three values of a and their corresponding detection thresholds. 
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Fig. 8. — Illustrating the Difference Between Confidence Intervals and Upper Limits. This figure is identical to 
Figure [1] with an additional solid vertical horizontal line showing the location of the upper limit computed with 
Pmin ~ 0.8. The legend denotes the true value of As, the assumed known value of As, and the computed upper limit, 
U. Note that unlike the confidence interval, which depends strongly on the number of observed source counts, ns, the 
upper limit is fixed once the detection threshold S* (which depends on As) and the minimum detection probability 
/3min are specified. 
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Fig. 9. — Interpreting Upper Bounds as Upper Limits. The three panels plot the probability or source 
detection for a source with As equal to the upper bound of Garwood's confidence intervals. Because the 
confidence intervals depend on ns, the detection probability, Pmin increases with ng. The three panels 
correspond to As = 1 (left), Xb — 3 (middle), and As = 5 (right), as in the columns of Figure [H All 
calculations were preformed with Tg = tb = r = 1. Because a source with intensity equal to the upper 
bound can have a detection probability as low as 20% or as high as 100%, the upper bound does not 
calibrate the maximum intensity that a source can have with appreciable probability of going undetected in 
any meaningful way. 
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Fig. 10. Conditional Coverage Probability of Confidence Interval Reported Only for Detected Sources. 
When confidence intervals are only reported for detected sources the coverage probability may be very 
different than the nominal level of the interval. This plots shows the true coverage of the 95% nominal 
intervals of Garwood (1936) when they are only reported for sources detected with a = 0.05 (with As = 3). 
For small values of A5 the coverage can be very low and as grows, the coverage probability converges to 
95%. Far fewer intervals contain As than one would expect given the nominal level. Confidence intervals 
must be reported regardless of ns and regardless of whether a source is detected. The jagged appearance of 
curve stems from the discrete nature of Poisson data. 
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Fig. 11. — Computing Upper Limits Based on the Probability of SNR Detection of a Source. The curves 
in each panel correspond to the probability of source detection as a function of As using an SNR detection 
threshold of S* = 3. The curves were computed with r = 1, rs = 1, = 1, and with = 10,20, and 
50 (dashed, dotted, and dash-dotted lines, respectively). Upper limits are computed by first adopting an 
acceptable probability for a source detection, and then computing the intercept on As of the l3{Xs) curves. 
The panels show the value of the upper limits for the different values of As for ^min = 0.5 (top), /Smin = 0.9 
(middle), and /3min = 0.95 (bottom). 
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Fig. 12. — Dependence of the Upper Limit on the Exposure Time. The shaded area to above and to the 
right of the curve is where the detection probabihty exceeds /Smin = 0.90. Thus the curves give the upper 
hmit (on the horizontal scale) as a function of the exposure time. The plot was made with As = 3, r = 5, 
Ts — tb, and a = 0.05 and shows how the upper limit generally decreases as the exposure time increases. 
Because of the discrete nature of Poisson data, the probability of type one error can not be set exactly equal 
to a. This results in the step function nature of a in Figure [5] and the non-monotonic decrease of the upper 
limit as a function of exposure time here. 



